Parallel SageAttention Inference #50
Merged
+208
−3
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DefTruth
changed the title
|
Thanks a lot! We well check the implementation and merge the PR. |
|
may need to install latest xDiT from source if your env already have FA>=2.7.0, i just make a hotfix to ensure ring flash attn forward compatible with lastest FA and thus will not run into an func launch error. also, the plug-and-play sage attention can only work with CFG parallelism for 2 GPUs now. |
|
I am busy these days and I will dive into it as soon as I can. |
This was referenced Dec 2, 2024
Closed
jt-zhang
pushed a commit
that referenced
this pull request
Dec 25, 2024
* relax nvcc version check for sm_89 * Update setup.py * workaround for distributed inference * Create parallel_sageattn_cogvideo.py * Create run_parallel.sh * Update README.md * Update core.py * Update run_parallel.sh * Update run_parallel.sh * Update core.py * Update README.md * Update core.py * Update core.py
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR add a small workaround that can make sage attention work compatible with distributed env, for example, xDiT which will launch by torchrun. Without this workaround, sage attention will run into illegal memory access error after first inference step in distributed env for multi gpus inference. This small workaround also make sage attention work compatible with torch.compile through non-fullgraph compile mode.
@jason-huang03